System and method for providing dynamic provisioning within a compute environment

ABSTRACT

The disclosure relates to systems, methods and computer-readable media for dynamically provisioning resources within a compute environment. The method aspect of the disclosure comprises A method of dynamically provisioning resources within a compute environment, the method comprises analyzing a queue of jobs to determine an availability of compute resources for each job, determining an availability of a scheduler of the compute environment to satisfy all service level agreements (SLAs) and target service levels within a current configuration of the compute resources, determining possible resource provisioning changes to improve SLA fulfillment, determining a cost of provisioning; and if provisioning changes improve overall SLA delivery, then re-provisioning at least one compute resource.

PRIORITY CLAIM

The present application is a continuation of U.S. patent applicationSer. No. 11/155,091, filed Jun. 17, 2005, which claims priority to U.S.Provisional Application No. 60/581,257 filed Jun. 18, 2004, the contentsof which are incorporated herein by reference.

BACKGROUND

1. Technical Field

The present disclosure relates to provisioning resources in a computeenvironment and more specifically to a system and method of providingdynamic provisioning for operating systems, applications and/or otherresources within the compute environment.

2. Introduction

The present disclosure relates to a system and method of managingresources in the context of a grid or cluster of computers. Gridcomputing can be defined as coordinated resource sharing and problemsolving in dynamic, multi-institutional collaborations. Many computingprojects require much more computational power and resources than asingle computer can provide. Networked computers with peripheralresources such as printers, scanners, I/O devices, storage disks,scientific devices and instruments, etc. can be coordinated and utilizedto complete a task.

Grid/cluster resource management generally describes the process ofidentifying requirements, matching resources to applications, allocatingthose resources, and scheduling and monitoring grid resources over timein order to run grid applications as efficiently as possible. Eachproject will utilize a different set of resources and thus is typicallyunique. In addition to the challenge of allocating resources for aparticular job, grid administrators also have difficulty obtaining aclear understanding of the resources available, the current status ofthe grid and available resources, and real-time competing needs ofvarious users.

Several general challenges exist when attempting to maximize resourcesin a grid. First, there are typically multiple layers of grid andcluster schedulers. FIG. 1 illustrates this point. A grid 100 generallycomprises a group of clusters or a group of networked computers. Thedefinition of a grid is very flexible and can mean a number of differentconfigurations of computers. The introduction here is meant to be verygeneral. The grid scheduler 102 communicates with a plurality of clusterschedulers 104A, 104B and 104C. Each of these cluster schedulerscommunication with a plurality of resource managers 106A, 106B and 106C.Each resource manager communicates with a series of compute resourcesshown as nodes 108A, 108B and 108C. These can be referred to as acluster or compute environment 110.

Second, local schedulers (which can refer to either the clusterschedulers 104A, 104B, 104C or the resource managers 106A, 106B, 106C)are closer to the specific resources 108 and may not allow gridschedulers 102 direct access to the resources. The grid level scheduler102 typically does not own or control the actual resources. Therefore,jobs are submitted from the high level grid-scheduler 102 to a local setof resources with no more permissions that the user would have. Thisreduces efficiencies.

Third, the heterogeneous nature of the shared resources causes areduction in efficiency. Without dedicated access to a resource, thegrid level scheduler 102 is challenged with the high degree of varianceand unpredictability in the capacity of the resources available for use.Most resources are shared among users and projects and each projectvaries from the other.

Fourth, the performance goals for projects differ. Grid resources areused to improve performance of an application but the resource ownersand users have different performance goals: from optimizing theperformance for a single application to getting the best systemthroughput or minimizing response time. Local policies can also play arole in performance. Several publications provide introductory materialregarding grid scheduling. See, e.g., Grid Resource Management, State ofthe Art and Future Trends, Jarek Nabrzyski, Jennifer M. Schopf, and JanWeglarz, Kluwer Academic Publishers, 2004; and Beowulf Cluster Computingwith Linux, edited by William Gropp, Ewing Lusk, and Thomas Sterling,Massachusetts Institute of Technology, 2003. The Beowulf ClusterComputing with Linux reference includes steps to creating a cluster.

Given the challenges associated with the compute environment,administrators have difficulty with regards to establishing operatingsystems and what operating systems are installed within a cluster. Inmany cases, clusters have a requirement for more than one operatingsystem, such as the Macintosh, AIX, Microsoft NT, Linux, and so forth.

The majority of cases, an administrator or a group of administrators ormanagers will determine before the fact what particular mixture of theseoperating systems will be needed to be installed on the cluster nodes.In addition to operating systems, the same challenges exist for otherresources within the cluster, such as software applications, memoryrequirements for each node, and other static or semi-static attributes.

These IT mangers and administrators must make a best estimate of thedistribution of their workload and then they set up the clusteraccordingly. Within a 64 node cluster, for instance, an administratorcan assign 48 nodes to one operating system, 12 nodes to anotheroperating system, and 4 more nodes to a third operating system. Theadministrator must anticipate what the workload will be. The problemwith this approach is that as the system comes on line, users begin tosubmit jobs according to their needs and not necessarily what wasconfigured by the operators and managers.

Load balancing issues can immediately exist between the variouspartitions which exist by virtue of the different operating systems.Partitions can relate to partitioning one of: operating systems, memory,disk space, a software application, a license or some other computeresource. One can find that the first operating system is under-utilizedwhile the second and third operating systems are heavily over-utilizedand there's nothing that can be done about it.

The cluster scheduler 104A, 104B, 104C simply does a matching policy tofigure out if a job comes in and requires a particular operating system,and which node or set of nodes are best for running the job. If thescheduler 104A, 104B, 104C attempts but cannot establish matches betweenjobs and nodes, it queues the job until a match is available when someother job completes.

What is needed is a way to allow the compute environment to map moredirectly to the resources requested by in the incoming jobs.

SUMMARY

Additional features and advantages of the invention will be set forth inthe description which follows, and in part will be obvious from thedescription, or can be learned by practice of the disclosure. Thefeatures and advantages of the disclosure can be realized and obtainedby means of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present disclosurewill become more fully apparent from the following description andappended claims, or can be learned by the practice of the disclosure asset forth herein.

The disclosure relates to systems, methods and computer-readable mediafor dynamically provisioning resources within a compute environment. Themethod aspect of the disclosure comprises A method of dynamicallyprovisioning resources within a compute environment, the methodcomprises analyzing a queue of jobs to determine an availability ofcompute resources for each job, determining an availability of ascheduler of the compute environment to satisfy all service levelagreements (SLAs) and target service levels within a currentconfiguration of the compute resources, determining possible resourceprovisioning changes to improve SLA fulfillment, determining a cost ofprovisioning; and if provisioning changes improve overall SLA delivery,then re-provisioning at least one compute resource. This is a lookingforward view of the queue and dynamically provisioning the environmentto meet the workload in the queue.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the disclosure briefly described above will berendered by reference to specific embodiments thereof which areillustrated in the appended drawings. Understanding that these drawingsdepict only typical embodiments of the disclosure and are not thereforeto be considered to be limiting of its scope, the disclosure will bedescribed and explained with additional specificity and detail throughthe use of the accompanying drawings in which:

FIG. 1 illustrates a prior art compute environment;

FIG. 2 illustrates a compute environment with a plurality of operatingsystems;

FIG. 3A illustrates a method embodiment of the disclosure;

FIG. 3B illustrates another method embodiment of the present disclosure;and

FIG. 4 illustrates a provisioning server associated with a clusterscheduler in an example of the application of the present disclosure.

DETAILED DESCRIPTION

Various embodiments of the disclosure are discussed in detail below.While specific implementations are discussed, it should be understoodthat this is done for illustration purposes only. A person skilled inthe relevant art will recognize that other components and configurationscan be used without parting from the spirit and scope of the disclosure.

The present disclosure addresses the deficiency in the prior art byproviding a dynamic provisioning approach to compute environmentmanagement. The present disclosure includes systems, methods andcomputer-readable media for providing dynamic, load-based look-aheadprovisioning of compute resources.

The present disclosure relates to managing resources within a computeenvironment. The environment can be operated by a hosting facility,hosting center, a virtual hosting center, data center, grid, clusterand/or utility-based computing environments and the like. The systemaspect of the disclosure comprises a computing device that operatessoftware that practices the steps of the disclosure to manage computeresources. There are many types of computing devices that are known tothose of skill in the art and that are acceptable as the systemembodiment of the disclosure. The computing device can be a singledevice or a plurality of connected computing devices that enable thedisclosure to be practiced. The software operating within the system iscomprised of computer program modules written in a computing language,such as the C programming language or any other suitable programminglanguage. The programming modules include all the necessary programmingto communicate with the compute environment (i.e., such as thecluster/grid) and both receive information about the compute resourceswithin the compute environment and also manage the reservation,provisioning and use of those compute resources.

The primary disclosure disclosed herein relates to the concept ofdynamically provisioning resources within the environment. Therefore,the system embodiment of the disclosure will include the various modulesthat practice the steps of the method embodiment of the disclosuredisclosed herein. The hardware used in such a system or computing devicefor the disclosure will include the basic known and future-developedhard components such as a central processor(s), a bus, memory, a harddisk, I/O devices such as modems or network cards, a display device(optional) and so forth. A system for managing compute resources withina compute environment may comprise means for provisioning resourceswithin a compute environment, the method comprising, means for analyzinga queue of jobs to determine an availability of compute resources foreach job, means for determining an availability of a scheduler of thecompute environment to satisfy all service level agreements (SLAs) andtarget service levels within a current configuration of the computeresources, means for determining possible resource provisioning changesto improve SLA fulfillment, means for determining a cost of provisioningand means for, if provisioning changes improve overall SLA delivery,re-provisioning at least one compute resource.

Availability or resources can be determined in terms of at least one of:a required OS, hardware architecture, network adapters, real memory,virtual memory, internal and external disk storage, and softwareapplications. Other factors can also be part of the analysis of whatresource is available. For example, availability can be constrained by aresource state, a reservation or other political policies.

The system can also include such components as means for instructing aprovisioning service to re-provision the compute environment if thehigher efficiency in the compute environment due to re-provisioning isgreater than the loss associated with re-provisioning and means forcontinuing to analyze the queue of jobs to study the workload if theloss associated with re-provisioning is higher than the efficiency inthe compute environment due to re-provisioning. The means for performingthis can be, as mentioned above, computer programmed modules within asoftware package that perform these steps and other method stepsdiscussed herein. An aspect of the analysis of jobs can involvedetermining a priority of jobs to compute environment objectives and/ora time frame in which jobs must complete to fulfill policies orrequirements such as those set forth in the SLAs. Jobs can be furtherconstrained in that only certain jobs are able to utilize or receiveprovisioning services. For example, jobs of a certain type or having acertain credential (from a particular group or a person on probation)can be prevented from receiving provisioning services.

Dynamic load-based look-ahead provisioning is a concept of utilizing apolicy engine to query an expected load on any given resource providerwhether it be a compute node with limited memory, network hardware withgiven bandwidth constraints, nodes with a given software productinstalled or particular operating system that is needed, softwarelicense issues, memory issues, disk space issues or other computeresource issues. Using the knowledge of current and future workloadneeds, the scheduler or other software module or modules provisionsadded resources to ensure that the limitations are overcome prior to theworkload attempting to be accomplished. An example of this would be ausing entity that needs eight nodes with application A on it for 12hours three days from a job submission. The reservation is set and aquery by a workload manager such as, for example, the Moab™ WorkloadManager, identifies that only six nodes are available with the requiredapplication and it sets up a software provisioning request (by sendinginstructions to a software provisioning manager), to add the neededsoftware to two additional nodes which it has reserved just prior towhen the reservation is to begin. In this way, the workload managerdynamically looks ahead, discovers based on current load which nodes areavailable and then provisions additional resources to meet the workloadneed.

A cluster scheduler typically operates on a server and communicates withother nodes via any known network. The basic configuration of clusterand grid schedulers, and their communication means with resourcemanagers and ultimately cluster resources such as nodes are known in theart.

The present disclosure involves enabling a cluster scheduler tointerface with a provisioning managing service to dynamically match theconfiguration of the resources to the current workload. To illustratehow this is done, the above example is discussed with reference to FIG.2. The architecture 200 shown in FIG. 2 includes a cluster 202 havingsixteen nodes 224. A job queue 210 currently contains a plurality ofjobs 212, 214, 216, 218, and 220. The partitioned operating systems, OS1204, OS2 206 and OS3 222 are shown. These operating systems each spaninto multiple nodes 224 of the cluster 202. Assume that there is anover-commitment of resources to OS2 and OS3 as shown by the jobs havingbeen assigned to their respective partitions for processing. In thisexample, jobs 212 and 214 are each assigned to OS1 204, job 216 isassigned to OS2, job 218 is assigned to OS3 and job 220 is assigned toprocess in both OS3 202 and OS2 206. The size of the partitionsillustrates the relative size and processing power of each of thepartitions of operating system. Since OS1 204 is larger, but has thesame amount of jobs assigned to it (two), we can assume that OS1 204 hasan over abundance of resources in comparison to OS2 206 and OS3 222.

In addition to referencing FIG. 2, the method embodiment of thedisclosure will be explained with reference to FIG. 3A. In response tothis incongruence, the present disclosure comprises analyzing a queue ofjobs 210 to determine an availability of compute resources for each job(302), determining an availability of a scheduler of the computeenvironment to satisfy all service level agreements (SLAs) and targetservice levels within a current configuration of the compute resources(304), determining possible resource provisioning changes to improve SLAfulfillment (306), determining a cost of provisioning (308) and ifprovisioning changes improve overall SLA delivery, then re-provisioningat least one compute resource (310).

The provisioning disclosure is independent of partitions whether theyare logical or physical. A partition can be bounded along the respectiveoperating systems. In other words, the various nodes within the clusterare logically partitioned so that each partition includes one of thethree operating systems. In other cases, compute resources can bepartitioned by the applications installed, the amount of memoryavailable, the amount of local disk available, licenses, etc. The samebasic approach works in all cases and one simply looks at the variousdimensions of partitioning and determines the percent of future workloadand the percentage of currently configured resources. The clusterscheduler then determines the amount of overhead required tore-provision the compute resources and determines whether the lossassociated with the re-provisioning process is less than the increasedefficiency which would result from the re-provisioned computeenvironment.

If the answer is yes, that re-provisioning increases efficiency, thenthe cluster manager proceeds to re-provision the cluster to bring itmore in alignment with the current workload and returns to the analysisof the job queue. If the loss due to re-provisioning is greater than theincreased efficiency from a re-provisioned compute environment, then themethod simply returns to the analysis step without re-provisioning. Theresulting cluster management approach is a cluster that is constantlyand dynamically updating itself to meet with the current workload. Thiscluster does not have the same inefficiencies associated with under orover-utilized resources over portioning boundaries.

This approach also saves on overhead costs of managers and administratorthat need to analyze and determine the ideal mixture of operatingsystem, memory requirements, software application requirements, etc. forconfiguration of clusters. The cluster basically adjusts itself to theworkload and the compute resources that are needed for optimalefficiency.

A cluster scheduler incorporating the present disclosure willcommunicate with a cluster provisioning service, such as Novell's RedCarpet, or IBM CSM, or an open source provisioning service. This featureis illustrated in FIG. 4. The cluster scheduler 402 will make thedetermination of how the node configuration should change and thencontact provisioning server 404 to request the change. The clusterscheduler can make the request in the follow way: please changeoperating system of node 226 from OS1 to OS3 and begin immediately.There can be offset and any kind of arbitrary instructions that areprovided to the provisioning service according to the analysis done bythe cluster scheduler of the workload and compute environment. As can beseen, the distribution of jobs and their assigned nodes within thecluster 202 from the job queue 210 is more evenly distributed.

In the context of the present disclosure, the term “cluster scheduler”can refer to a number of managing applications within a computingenvironment such as a cluster or a grid. For example, this term canapply to a resource manager, a cluster scheduler, a grid scheduler, aworkload manager, a cluster or a grid monitor, a cluster manager and soforth. There are a number of software applications in the gridenvironment that can manage or schedule work at various layers of thenetwork. The term “cluster scheduler” can refer to any of thesecomponents at the various layers of a grid or cluster.

In another aspect of the disclosure shown in FIG. 3B, when the clusterscheduler makes a determination that it should re-provision the cluster,it can analyze the current workload schedule associated with the node ornodes that need to be re-provisioned (330). These nodes may or may notbe reserved. The cluster scheduler will then select the at least onenode that can be re-provisioned with minimal impact (332) and when theat least one node becomes available, instructing a provisioning serviceto re-provision the at least one node (334). This process may or may notinvolve creating a reservation for the one or more nodes. Such areservation is shown as feature 406 in FIG. 4. Then, when the node(s)become available and there is no longer a workload running on them, thecluster scheduler 402 contacts the provisioning manager or provisioningserver 404 and instructs the server 404 to change the state of(re-provision) the cluster. When the cluster scheduler 402 receivesconfirmation from the provisioning server 404 indicating that the changeof state was successful, the cluster scheduler 402 releases the at leastone node back to general scheduling and jobs can be started on thenode(s) (336).

It is also noted that as shown in FIG. 4, the provisioning server 404 isa separate process or software entity from the cluster scheduler 402.This is a preferable way to perform these steps but it is alsocontemplated that the provisioning service would be incorporated as partof the cluster scheduler 402. It is immaterial to the present disclosurewhether these processes are performed by two different softwareapplications or one. Therefore, where the disclosure relates toproviding instructions to a provisioning service to achieve there-provisioning process, this includes the cluster scheduler instructingan internal provisioning service to carry out these processes.

If the cluster scheduler 402 determines that it is too expensive to do afull provision, an alternate step shown in FIG. 3A is to determinewhether a partial re-provisioning can occur (314). If yes, then themethod proceeds with the partial provisioning. If the analysis of apartial provisioning further determines that the overhead for thatpartial provisional is higher than the efficiencies gained, then theprocess returns to the initial analysis step (308).

As discussed above, the cost of re-provisioning is considered in thedecision of whether or not to re-provision. The overhead associated withprovisioning relates to the wasted resources that are not allocated toany job. In some cases, where the re-provisioning to occur can beconsidered “light-weight” and quick and easy to evaluate and complete,the cluster scheduler can dynamically take into account short termhistorical information (in whole or in part) to determine whether tocontinue with the re-provisioning. This historical information can bebased on performance of the cluster, based on failure rates, based onquantity or loads according to job submissions, or any type of usefulinformation about the compute resources and job submissions. Anycombination of historical information can be utilized. The short termcan include information over the past 5 hours (or any desired timeframe) and whether resources were relatively over or underutilized giventhe workload for that period of time plus the current back-log and/orcurrent workload. Determining if provisioning changes would improve theoverall SLA delivery can be subject to provisioning costs in both longterm and short term.

Instructions provided to the provisioning service can be modified basedon analyzed historical information and the modification can involve suchthings as speeding up or slowing down the re-provisioning of the computeenvironment based on the historical information.

Another case relates to the ability of the cluster scheduler to recordlonger term information and determine not only what are the losses dueto the operating system level partitioning (or whatever other resourceis partitioned) that are going on within the current workload, but alsohow long this inefficiency been existed. For example, the system candetermine what the losses have been over the last two weeks, the lastthree weeks, or any desired time frame. Those losses can be incorporatedinto the re-provisioning decision as well as the instructions to theprovisioning service. This historical view can modify there-provisioning timing and decision wherein the adjustments andmodification can be made more slowly or more quickly. The instructionscan also be modified to re-provision resources in a different order orin a different manner taking into account this historical information.In such cases, by extending the view the cluster scheduler can oftenmake the costs of provisioning less than the cost of losses associatedwith the system.

As an example of a light-weight provisioning decision based onback-logged workload, consider if the re-provisioning relates tochanging memory. Provisioning the change in memory takes about 30seconds. If the evaluation of the current workload indicates that it isbacklogged for the next 8 hours and the cluster would run slightly moreefficiently with the memory change, then that decision can be madeimmediately without having to care about historical information becausethe cost of provisioning is so low. In another case, the cost ofprovisioning an operating system is on the order of six hours and thereis a high failure probability the system will probably want toincorporate historical information, review what has previously happened,how the workload has adjusted over the last two weeks, what kind oflosses occurred over the last couple of weeks, and then compare thoselosses on a per node basis, with the losses to re-provision on a pernode basis. A threshold value can be set which if the cost ofprovisioning passes that threshold value, then the historicalinformation will be incorporated. Therefore, the analyzed backlog whichincludes a work backlog analysis can be used to determine whether theloss associated with the re-provisioning justifies the overhead amountin comparison to a higher efficiency of the compute environment whichwould result from the re-provisioning.

As suggested above, an aspect of the disclosure involves analyzingfailure probabilities for the re-provisioning and including the failureprobability analysis in the determination of whether the cost associatedwith re-provisioning justifies the overhead amount in comparison to ahigher efficiency of the compute environment which would result from there-provisioning.

For example, the historical information allows the cluster scheduler toanalyze the cluster along soft partitions such as the operatingpartition or memory partition (any type of partition), and decide thatwithin a given partition OS1 is only running at 90% utilization, whileOS2 and OS3 have both had 99% utilization. In that case, the systemknows that there has been 10% available utilization over the last 2weeks, and it can look at that historical information to assist indetermining the cost to the cluster over time. In this regard, thisapproach does not directly identify how successful the provisioning hasbeen, but relates to how long the workload configuration has causedunder-commitment of OS1 and over or under-commitment of OS2 and OS3.

Another aspect of the present disclosure is that it is not limited orconstrained to any particular compute resource, such as a node. Theresources that can be re-provisioned include the operating systems, enduser applications installed on the nodes, memory upgrades, local diskspace available, or network provisioning with guaranteed bandwidth on aper node basis. The provisioning can also be for the number ofprocessors assigned to a particular job, node or partition. There-provisioning may also be for any other resource such as bandwidth,licenses, network resources, router usage, and so forth.

If the possible resource provisioning improves SLA fulfillment, then themethod further comprises generating a schedule of times to provisioneach selected resource. Then re-provisioning compute resources occurs astheir scheduled time arrives. These scheduled times may or may not berelated to a reservation of the resources. In the reservation context,the disclosure would comprise creating a reservation for the at leastone node to be re-provisioned. A group of nodes can also be reserved orabsent a reservation, identified for re-provisioning. The nodes can bere-provisioned at the same time or can be re-provisioned in a piecemealfashion depending on other factors such as workload usage. This canprovide a reduced impact or cost for the re-provisioning whereindividual nodes are re-provisioned as they become available or based onsome factor such as when each node's current configuration or usagejustifies re-provisioning. Whereas other nodes within the identifiedgroup may not be in the same position to be re-provisioned.

After each node is re-provisioned it is release back into a generalavailability mode. The determination of when to release a node afterre-provisioning can be independent of whether a provisioning service ora provisioning manager has the ability to report back success or failureof the re-provisioning step. There are various ways of more efficientlyreleasing the node back to general availability depending on the contextand manner of its identification and re-provisioning process. Forexample, the scheduler can release the at least one compute resource (ornode) back to a general availability occurs according to one of thefollowing time frames: when a provisioning manager confirms that there-provisioning is complete; if a re-provisioning scheduled time is set,ahead of the scheduled time if the provisioning manager confirms thatthe re-provisioning is complete; or if a re-provisioning scheduled timeis set, after the scheduled time if the provisioning manager confirmsthat the re-provisioning is not complete or failed. If reservations areused to control resource access and schedule resource provisioning intime, then the scheduler can release resources from reservations priorto a reservation end time if the provisioning manager confirmssuccessful completion of re-provisioning. The scheduler can extend thereservation while the provisioning manager confirms failure ofre-provisioning.

Embodiments within the scope of the present disclosure can also includecomputer-readable media for carrying or having computer-executableinstructions or data structures stored thereon. Such computer-readablemedia can be any available media that can be accessed by a generalpurpose or special purpose computer. By way of example, and notlimitation, such computer-readable media can comprise non-transitorycomputer-readable media including RAM, ROM, EEPROM, CD-ROM or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to carry or store desiredprogram code means in the form of computer-executable instructions ordata structures. When information is transferred or provided over anetwork or another communications connection (either hardwired,wireless, or combination thereof) to a computer, the computer properlyviews the connection as a computer-readable medium. Thus, any suchconnection is properly termed a computer-readable medium. Combinationsof the above should also be included within the scope of thecomputer-readable media.

Computer-executable instructions include, for example, instructions anddata which cause a general purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions. Computer-executable instructions also includeprogram modules that are executed by computers in stand-alone or networkenvironments. Generally, program modules include routines, programs,objects, components, and data structures, etc. that perform particulartasks or implement particular abstract data types. Computer-executableinstructions, associated data structures, and program modules representexamples of the program code means for executing steps of the methodsdisclosed herein. The particular sequence of such executableinstructions or associated data structures represents examples ofcorresponding acts for implementing the functions described in suchsteps.

Those of skill in the art will appreciate that other embodiments of thedisclosure can be practiced in network computing environments with manytypes of computer system configurations, including personal computers,hand-held devices, multi-processor systems, microprocessor-based orprogrammable consumer electronics, network PCs, minicomputers, mainframecomputers, and the like. Embodiments can also be practiced indistributed computing environments where tasks are performed by localand remote processing devices that are linked (either by hardwiredlinks, wireless links, or by a combination thereof) through acommunications network. In a distributed computing environment, programmodules can be located in both local and remote memory storage devices.

Although the above description can contain specific details, they shouldnot be construed as limiting the claims in any way. Other configurationsof the described embodiments of the disclosure are part of the scope ofthis disclosure. Accordingly, the appended claims and their legalequivalents should only define the disclosure, rather than any specificexamples given.

I claim:
 1. A method comprising: determining, at a current time, anavailability of compute resources within a compute environment for eachworkload in a queue of workload based on scheduled times of computeresources established for the each workload to yield an analysis, theeach workload to consume scheduled compute resources at a future time,which is later than the current time, according to the scheduled times;based on the analysis, determining an ability of the compute environmentto satisfy a service level agreement associated with the each workloadin the queue of workload based on a current configuration of the computeenvironment; determining possible workload scheduling changes in thecompute environment to improve service levels for at least one workloadin the queue of workload; determining a cost of provisioning computeresources associated with the possible workload scheduling changes; andif the possible workload scheduling changes improve service levelswithin the compute environment, taking into account the cost,rescheduling the at least one workload in the compute environment priorto all workload in the queue of workload consuming compute resources inthe compute environment.
 2. The method of claim 1, further comprisingdetermining if the possible workload scheduling changes improve servicelevels within the compute environment based on provisioning costs inboth a long term and a short term.
 3. The method of claim 1, furthercomprising provisioning compute resources as a respective scheduled timearrives for the at least one workload.
 4. The method of claim 1, whereinavailability is determined in terms of one of: a required operatingsystem, hardware architecture, network adapters, real memory, virtualmemory, internal and external disk storage, and a software application.5. The method of claim 1, wherein the availability is furtherconstrained by one of: a resource state, a reservation and a politicalpolicy.
 6. The method of claim 1, further comprising determining apriority of workload to compute environment objectives and a time framein which workload must complete to fulfill service level agreements. 7.The method of claim 1, wherein only certain workload in the queue ofworkload are enabled to receive provisioning services.
 8. The method ofclaim 1, further comprising, if the compute environment will berescheduled: analyzing a workload schedule associated with a node to berescheduled; selecting at least one node that can be scheduled with aminimal impact; and when the at least one node becomes available,instructing a scheduling service to schedule the at least one node. 9.The method of claim 8, wherein scheduling further comprises creating areservation for the at least one node and provisioning the at least onenode.
 10. The method of claim 9, wherein provisioning the at least onenode further comprises provisioning a group of nodes within the computeenvironment.
 11. The method of claim 9, wherein provisioning the atleast one node further comprises provisioning individual nodes piecemealwithin the compute environment.
 12. The method of claim 11, whereinprovisioning the individual nodes piecemeal within the computeenvironment is based on workload within the compute environment.
 13. Themethod of claim 1, further comprising releasing the compute resourcesback to a general availability.
 14. The method of claim 13, wherein ascheduler releases the compute resources back to a general availabilityaccording to one of the following time frames: when a provisioningmanager confirms that a provisioning is complete; if a provisioningscheduled time is set, ahead of the provisioning scheduled time if theprovisioning manager confirms that the provisioning is complete; and ifa re-provisioning scheduled time is set, after the re-provisioningscheduled time if the provisioning manager confirms that theprovisioning is one of complete and failed.
 15. The method of claim 13,wherein reservations are used to control resource access and scheduleresource provisioning in time.
 16. The method of claim 15, wherein ascheduler will release resources from reservations prior to areservation end time if a provisioning manager confirms successfulcompletion of provisioning.
 17. The method of claim 16, wherein thescheduler will extend the reservation while the provisioning managerconfirms failure of provisioning.
 18. The method of claim 1, furthercomprising: analyzing historical information of the compute environment;and determining whether to schedule the compute environment based atleast in part on the historical information.
 19. The method of claim 18,wherein the historical information is one of compute environmentperformance information, workload-submission information, and failurerate information.
 20. A non-transitory computer-storage medium havingstored therein instructions which, when executed by a processor, causethe processor to perform operations comprising: determining, at acurrent time, an availability of compute resources within a computeenvironment for each workload in a queue of workload based on scheduledtimes of compute resources established for the each workload to yield ananalysis, the each workload to consume scheduled compute resources at afuture time, which is later than the current time, according to theschedules times; based on the analysis, determining an ability of thecompute environment to satisfy a service level agreement associated withthe each workload in the queue of workload based on a currentconfiguration of the compute environment; determining possible workloadscheduling changes in the compute environment to improve service levelsfor at least one workload in the queue of workload; determining a costof provisioning compute resources associated with the possible workloadscheduling changes; and if the possible workload scheduling changesimprove service levels within the compute environment, taking intoaccount the cost, rescheduling the at least one workload in the computeenvironment prior to all workload in the queue of workload consumingcompute resources in the compute environment.
 21. A system comprising: aprocessor; and a computer-readable storage medium having stored thereininstructions which, when executed by the processor, cause the processorto perform operations comprising: determining, at a current time, anavailability of compute resources within a compute environment for eachworkload in a queue of workload based on scheduled times of computeresources established for the each workload to yield an analysis, theeach workload to consume scheduled compute resources at a future time,which is later that the current time, according to the schedules times;based on the analysis, determining an ability of the compute environmentto satisfy a service level agreement associated with the each workloadin the queue of workload based on a current configuration of the computeenvironment; determining possible workload scheduling changes in thecompute environment to improve service levels for at least one workloadin the queue of workload; determining a cost of provisioning computeresources associated with the possible workload scheduling changes; andrescheduling the at least one workload in the compute environment priorto all workload in the queue of workload consuming compute resources inthe compute environment, if the possible workload scheduling changesimprove service levels within the compute environment, taking intoaccount the cost.